Virtual reality interaction method, apparatus and system

ABSTRACT

A virtual reality interaction method includes: rendering content including an object; tracking the object in the content; and performing gesture tracking for supporting gesture control on the object; wherein an application programing interface API framework is provided to allow a user to define at least one of the content, the tracking module and the interactivity module.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation-in-Part Application of U.S. patentapplications Ser. No. 15/794,016 filed on Oct. 26, 2017, which claimsthe benefit of U.S. Provisional Patent Applications No. 62/546,551 filedon Aug. 17, 2017, the entire contents of which are incorporated hereinby reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present disclosure relates to the field of virtual realitytechnologies, and more particularly, to a virtual reality interactionmethod, apparatus and system for accurately simulating an interactiveaction between a user and a real object.

2. Description of the Prior Art

With the development of technologies, virtual reality technologies arewidely used in life.

In the related art, when a user expects to get a lifelike interactiveexperience on a real object in a virtual environment, in most cases theuser needs to wear a positioner on a hand or need other specialelectronic instruments or manipulators. When the user needs to interactwith the real object in a virtual reality environment, for example,reading a virtual book, the related art cannot provide a convenientoperation mode for the user to read the virtual book in a virtual world.

It is to be noted that the above information disclosed in thisBackground section is only for enhancement of understanding of thebackground of the present disclosure and therefore it may containinformation that does not form the prior art that is already known to aperson of ordinary skill in the art.

SUMMARY OF THE INVENTION

As described above, when the user reads the virtual book, in the relatedart, it is impossible to distinguish an operation relationship bycombining a position between a hand action of the user and a virtualobject to achieve convenient operation, for example, page flipping.

Therefore, an objective of the present disclosure is to provide avirtual reality interaction method, apparatus and system for accuratelysimulating an interactive action between a user and a real object toovercome, at least to a certain extent, one or more problems caused bylimitation and defects of related technologies.

According to a first aspect of embodiments of the present disclosure,there is provided a virtual reality interaction method, which includes:continuously acquiring image information in front of a user; recognizinga preset interactive object by recognizing a preset identifier in theimage information, a surface of the preset interactive object having oneor more of the preset identifiers; recognizing a hand action of the useraccording to the image information; and simulating an interactive actionof the user on the preset interactive object in a virtual realityenvironment according to the hand action and a position of the presetinteractive object.

According to a second aspect of the embodiments of the presentdisclosure, there is provided a virtual reality interaction apparatus,including:

-   -   an image capturing module, configured to continuously acquire        image information in front of a user;    -   an identifier recognizing module, configured to recognize a        preset interactive object by recognizing a preset identifier in        the image information, a surface of the preset interactive        object having one or more of the preset identifiers;    -   a gesture recognizing module, configured to recognize a hand        action of the user according to the image information; and    -   an image output module, configured to simulate an interactive        action of the user on the preset interactive object in a virtual        reality environment according to the hand action and a position        of the preset interactive object.

According to a third aspect of the embodiments of the presentdisclosure, there is provided a virtual reality interaction system,including: a virtual reality display apparatus, configured to display avirtual reality image for a user; a preset interactive object,configured to assist in recognizing an interactive action of the user;and the virtual reality interaction apparatus as described above.

According to a fourth aspect of the embodiments of the presentdisclosure, there is provided an electronic device, including: one ormore processors; and a storage apparatus, configured to store one ormore programs. When the one or more programs are executed by the one ormore processors, the one or more processors are caused to implement theabove virtual reality interaction method.

According to a fifth aspect of the embodiments of the presentdisclosure, there is provided a computer readable storage medium, whichstores a computer program. When the computer program is executed by theprocessor, the virtual reality interaction method is implemented.

According to a sixth aspect of the embodiments of the presentdisclosure, there is provided a virtual reality interaction method,which includes: acquiring image information of a preset interactiveobject, a surface of the preset interactive object having a presetidentifier; acquiring at least one parameter of the preset interactiveobject by recognizing the image information corresponding to the presetidentifier; and executing a preset interactive operation according tothe at least one parameter.

According to a seventh aspect of the embodiments of the presentdisclosure, there is provided a virtual reality interaction apparatus,including: an image capturing module, configured to acquire imageinformation of a preset interactive object, a surface of the presetinteractive object having a preset identifier; a parameter acquiringmodule, configured to acquire at least one parameter of the presetinteractive object by recognizing the image information corresponding tothe preset identifier; and an interaction execution module, configuredto execute a preset interactive operation according to the at least oneparameter.

According to an eighth aspect of the embodiments of the presentdisclosure, there is provided another virtual reality interactionmethod, including: rendering content including an object; tracking theobject in the content; and performing gesture tracking for supportinggesture control on the object; wherein an application programinginterface API framework is provided to allow a user to define at leastone of the content, the tracking module and the interactivity module.

According to an ninth aspect of the embodiments of the presentdisclosure, there is provided a virtual reality interaction system,including: a content display configured to render content including anobject; a tracking module configured to track the object in the content;an interactivity module configured to perform gesture tracking forsupporting gesture control on the object; and an application programinginterface API framework configured to allow a user to define at leastone of the content, the tracking module and the interactivity module.

According to a tenth aspect of the embodiments of the presentdisclosure, there is provided an electronic device, including: one ormore processors; and a storage apparatus, configured to store one ormore programs. When the one or more programs are executed by the one ormore processors, the one or more processors are caused to implement theabove virtual reality interaction method.

According to an eleventh aspect of the embodiments of the presentdisclosure, there is provided a computer readable storage medium, whichstores a computer program. When the computer program is executed by theprocessor, the above virtual reality interaction method is implemented.

According to the virtual reality interaction method of the presentdisclosure, a position of a preset interactive object is determined anda hand action of a user is determined by recognizing a preset identifieron the preset interactive object, and the preset interactive object anda relative position relationship between the hand of the user and thepreset interactive object are accurately positioned. In this way, aninteractive action of the user on a real object is accurately simulatedin a virtual reality environment, and user experience of virtual realityis improved.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the present disclosure.

These and other objectives of the present invention will no doubt becomeobvious to those of ordinary skill in the art after reading thefollowing detailed description of the preferred embodiment that isillustrated in the various figures and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings herein, which are incorporated in andconstitute a part of this specification, illustrate embodimentsconsistent with the present disclosure and, together with thedescription, serve to explain the principles of the present disclosure.Apparently, the accompanying drawings in the following description showmerely some embodiments of the present disclosure, and persons ofordinary skill in the art may still derive other drawings from theseaccompanying drawings without creative efforts.

FIG. 1 schematically illustrates a flowchart of a virtual realityinteraction method according to an exemplary embodiment of the presentdisclosure;

FIG. 2 schematically illustrates a schematic diagram of a virtualreality helmet according to an exemplary embodiment of the presentdisclosure;

FIG. 3 schematically illustrates a schematic diagram of a presetidentifier according to an exemplary embodiment of the presentdisclosure;

FIG. 4 schematically illustrates a schematic diagram of a single-pagebooklet according to an exemplary embodiment of the present disclosure;

FIG. 5 schematically illustrates a schematic diagram of a multipagebooklet according to an exemplary embodiment of the present disclosure;

FIG. 6 schematically illustrates a schematic diagram in which a handshields an upper identifier of a booklet according to an exemplaryembodiment of the present disclosure.

FIGS. 7-11 schematically illustrate schematic diagrams of handcalibration procedures according to exemplary embodiments of the presentdisclosure.

FIGS. 12-15 schematically illustrate schematic diagrams of procedures ofstarting gesture commands according to exemplary embodiments of thepresent disclosure;

FIGS. 16-21 schematically illustrate schematic diagrams of simulating auser's flipping over procedures according to exemplary embodiments ofthe present disclosure.

FIGS. 22-27 schematically illustrate schematic diagrams of procedures ofselecting books by the user according to exemplary embodiments of thepresent disclosure;

FIG. 28 schematically illustrates a block diagram of a virtual realityinteraction apparatus according to an exemplary embodiment of thepresent disclosure;

FIG. 29 schematically illustrates a flowchart of another virtual realityinteraction method according to an exemplary embodiment of the presentdisclosure.

FIG. 30 schematically illustrates a schematic diagram of a general stateof a booklet according to an exemplary embodiment of the presentdisclosure;

FIG. 31 schematically illustrates a schematic diagram in which a rightpage of a booklet is incurved according to an exemplary embodiment ofthe present disclosure;

FIG. 32 schematically illustrates a schematic diagram in which a leftpage of a booklet is incurved according to an exemplary embodiment ofthe present disclosure;

FIG. 33 schematically illustrates a schematic diagram in which a rightpage of another booklet is incurved according to an exemplary embodimentof the present disclosure.

FIG. 34 schematically illustrates a schematic diagram of a normal vectorof a tag of a booklet according to an exemplary embodiment of thepresent disclosure.

FIG. 35 schematically illustrates a schematic diagram of a first normalvector in a first identifier and a second normal vector in a secondidentifier according to an exemplary embodiment of the presentdisclosure.

FIG. 36 schematically illustrates a schematic diagram in which a rightpage bends and deforms with time according to an exemplary embodiment ofthe present disclosure.

FIG. 37 schematically illustrates a schematic diagram in which anincluded angle between a first normal vector and a second normal vectorbased on FIG. 36 varies with time.

FIG. 38 schematically illustrates a schematic diagram in which theincluded angle curve as shown in FIG. 37 falls within a predefinedregion according to an embodiment of the present disclosure.

FIG. 39 schematically illustrates another schematic diagram in which anincluded angle curve falls within a predefined region according to anembodiment of the present disclosure.

FIG. 40 schematically illustrates a schematic diagram of executing anoperation of quick flip forward according to an embodiment of thepresent disclosure.

FIG. 41 schematically illustrates a schematic diagram of an includedangle curve when executing the operation of quick flip forward accordingto an embodiment of the present disclosure.

FIG. 42 schematically illustrates a schematic diagram of anotherincluded angle curve when executing the operation of quick flip forwardaccording to an embodiment of the present disclosure;

FIG. 43 schematically illustrates a block diagram of another virtualreality interaction apparatus according to an exemplary embodiment ofthe present disclosure;

FIG. 44 schematically illustrates a schematic structural diagram of anelectronic device according to an exemplary embodiment of the presentdisclosure;

FIG. 45 schematically illustrates a flowchart of a virtual realityinteraction method according to another embodiment of the presentdisclosure; and

FIG. 46 schematically illustrates a block diagram of a virtual realityinteraction system according to an exemplary embodiment of the presentdisclosure.

DETAILED DESCRIPTION

Now, exemplary embodiments will be described more comprehensively withreference to the drawings. However, the exemplary embodiments can beembodied in many forms and should not be construed as limited to theembodiments set forth herein; rather, these embodiments are provided sothat this disclosure will be made thorough and complete, and the conceptof exemplary embodiments will be fully conveyed to those skilled in theart. Furthermore, the described features, structures, or characteristicsmay be combined in any suitable manner in one or more embodiments. Inthe following description, numerous specific details are provided toprovide a thorough understanding of the embodiments of the presentdisclosure. Those skilled in the art will recognize, however, that thetechnical solution of the present disclosure may be practiced withoutone or more of the specific details described, or that other methods,components, materials, etc. may be employed. In other instances,well-known technical solutions are not shown or described in detail toavoid obscuring aspects of the present disclosure.

In addition, the accompanying drawings are merely exemplary illustrationof the present disclosure, and are not necessarily drawn to scale. Thesame reference numerals in the drawings denote the same or similarparts, and thus repeated description thereof will be omitted. Some blockdiagrams shown in the figures are functional entities and notnecessarily to be corresponding to a physically or logically individualentities. These functional entities may be implemented in software form,or implemented in one or more hardware modules or integrated circuits,or implemented in different networks and/or processor apparatuses and/ormicrocontroller apparatuses.

A detailed description of the exemplary embodiments of the presentdisclosure will be made in the following with reference to theaccompanying drawings.

FIG. 1 is a flowchart of a virtual reality interaction method accordingto the present disclosure. Referring to FIG. 1, the virtual realityinteraction method 100 may include following steps:

Step S102: image information in front of a user is continuouslyacquired;

Step S104: a preset interactive object is recognized by recognizing apreset identifier in the image information, a surface of the presetinteractive object having one or more of the preset identifiers;

Step S106: a hand action of the user is recognized according to theimage information; and

Step S108: an interactive action of the user on the preset interactiveobject is simulated in a virtual reality environment according to thehand action and a position of the preset interactive object.

A highly immersive mixed reality experience may be constructed accordingto a position and an orientation of the preset interactive objectcalculated in the above steps, a virtual content associated with thepreset identifier, a gesture and a position of the user, and aninteractive result of the gesture in the system.

According to the virtual reality interaction method 100 of the presentdisclosure, a position of a preset interactive object is determined byrecognizing a preset identifier on the preset interactive object and ahand action of a user is determined, and the preset interactive objectand a relative position relationship between the hand of the user andthe preset interactive object are accurately positioned. In this way, aninteractive action of the user on a real object is accurately simulatedin a virtual reality environment, and user experience of virtual realityis improved.

The virtual reality interaction method is described in detail below withreference to specific embodiments.

FIG. 2 schematically illustrates a schematic diagram of a virtualreality (VR) helmet according to an exemplary embodiment of the presentdisclosure. The virtual reality helmet may be, for example, ahead-mounted display (HMD). The helmet 200 may include a virtual realitydisplay apparatus 202 and an image capturing apparatus (for examplecamera) 204.

The above method 100 maybe implemented by wearing the virtual realityHMD. The virtual reality HMD of the present disclosure includes one ormore image capturing apparatuses, which may be, for example, cameras.The camera maybe mounted on the HMD in an internal or external form tocapture an environment in front of the HMD so as to continuously acquireimage information in front of a user.

The HMD generally is provided with two display screens, respectivelyarranged in front of eyes, used to display image output results of avirtual reality system. In this system, one processor or other relatedmodules may also be provided to perform, in real time, calculation ofeach module and display rendering results on the display screens of theHMD. The processor and the other related modules may be built in theHMD, or may be an external apparatus connected to the HMD, either wiredor wirelessly, to transmit data.

In an ordinary virtual reality system, after the user wears the HMD, aline of sight of the user may almost be shielded wholly to isolate anexternal light source. In addition, in the virtual reality system of theHMD with a camera according to the present disclosure, the camera cancapture an external environment in front of the user. The cameramentioned in this embodiment may be an RGB channel-based camera, anRGB-D camera, a binocular camera or other cameras common in VR and ARsystems.

To provide the user with an experience with mixed reality, objects infront of the HMD or available for interaction should be displayed on thedisplay screen of the HMD in a virtual object manner. According to onemethod, by performing computer vision algorithm calculation on an imagecaptured by the camera of the HMD, the position, the orientation and theposture of the preset interactive object are tracked. According toanother method, a sensor is attached to the preset interactive object.Under the premise of independent of the HMD and an additional sensor ofthe preset interactive object, the virtual reality interaction method100 may provide a mixed reality experience via the first method.

FIG. 3 schematically illustrates a schematic diagram of a presetidentifier according to an exemplary embodiment of the presentdisclosure. Referring to FIG. 3, the preset identifier may be atwo-dimensional identifier including a two-dimensional code, a barcodeor other two-dimensional patterns, or may be other identifiers that canbe recognized by the virtual reality image capturing apparatus andanalyzed by the processor afterward. Each preset identifier has anindependent identity (ID) to be differentiated from other presetidentifiers.

To improve the robustness of the system in tracking the position, theorientation and different postures of the preset interactive object, thepreset identifier may be printed on the surface of the presetinteractive object. Combination of the preset identifier with the presetinteractive object may include printing or adhering an image (forexample, AprilTag) onto the preset interactive object, or adhering theimage to the surface of the preset interactive object in any way.

The preset interactive object may be, for example, a booklet or book, oran object, similar in shape to a booklet or book, made from a hard orsoft material, which may have one or more pages and also may be madeinto a foldable form. The booklet may serve as a target tracking object,and display data according to different designs and requirements. Basedon a fact that the preset identifier may be shielded by other objects(for example, a hand), and in order to improve the accuracy of trackingthe position of the object, a plurality of preset identifiers may beprinted on or adhered to each page of the booklet. In an embodiment, thebooklet may have a single page and is printed with one or more presetidentifiers. In another embodiment, the booklet may be a flippable pagewith a front side and a back side, and both sides are printed with oneor more preset identifiers. In another embodiment, the booklet may havetwo or more foldable pages, each of which is printed with one or morepreset identifiers. For example, FIG. 4 schematically illustrates aschematic diagram of a single-page booklet according to an exemplaryembodiment of the present disclosure; and FIG. 5 schematicallyillustrates a schematic diagram of a multipage booklet according to anexemplary embodiment of the present disclosure.

Positions and orientations of one or more preset identifiers may bedetermined according to images of the one or more preset identifiers inacquired image information, and the position and the orientation of thebooklet may be calculated according to relative positions of the presetidentifiers with respect to the booklet. Data of the relative positionmay be predefined, or may be corresponding information searched from adatabase according to a unique code of the preset identifier.

A virtual booklet may be displayed in the HMD according to the data onthe basis of the position and the orientation of the obtained booklet.In order to further improve the immersion, in addition to the positionand the orientation, a real size of the booklet and a virtual size ofthe booklet displayed in the helmet may be corresponding according to aratio of 1 to 1, or may be corresponding according to a fixed ratio orother nonlinear ratios allowing the user to feel a high immersion.

If the booklet has multiple pages, in an embodiment, by tracking thepreset identifiers, when the user flips the booklet to a next page, thevirtual booklet may correspondingly display a content of the page. Inthis scenario, the number of pages of the virtual booklet needs to beconsistent with that of pages of a physical booklet. Nevertheless, inanother embodiment, the requirement for the number of pages of thephysical booklet may be lowered via gesture recognition. When the userexecutes a page flipping gesture, page flipping of the virtual bookletmay be triggered, and a process and a result of the page flipping aredisplayed. Therefore, this method may support the user to simulate tobrowse a virtual booklet with infinite pages in the virtual environmentby operating the entity booklet with at least one page.

In addition to displaying characters and picture contents in the entitybooklet generally, a multimedia content (such as animation, video, musicand sound), a 3D content and a 360 virtual reality content may bedisplayed in the virtual booklet, for example, displayed in the virtualbooklet or displayed in a position, corresponding to the virtualbooklet, in virtual space. The content is displayed or the interactionis performed by means of a gesture or an interactive operation of othercontrol devices on the booklet or other objects in the system.

FIG. 6 schematically illustrates a schematic diagram in which a handshields an upper identifier of a booklet according to an exemplaryembodiment of the present disclosure. A unique code (ID) of the presetidentifier on the booklet may be read and the position and theorientation of the booklet relative to the camera may be calculated outby means of a photo acquired by the camera.

By using a computer vision tracking algorithm in this module, the systemcan track a target object appearing in the camera, for example, thebooklet or other similar objects. The tracking algorithm extracts keyfeatures of the object, for example, a color, an edge, a texture and soon. This is a topic which has been studied in depth. To provide ahigh-quality mixed reality system, the tracking algorithm of this moduleneeds to reach a high accuracy (for example a millimeter class), hashigh robustness in different types of objects (such as shape, color andsurface) and different peripheral environments (such as lamplight andbackground), and calculate a result highly in real time (for example, 60frames/s).

To avoid the various problems as described above, a known pattern isadhered to or printed on the preset interactive object. According tothis known pattern, the physical size of the pattern and parameters ofthe camera (such as a focal length and a lens type), the position andthe orientation of the pattern relative to the camera may be calculatedout using geometric graphics, and then the position and the orientationof the object relative to the camera may be calculated out through theposition and the orientation of the pattern relative to the presetinteractive object.

An example of the pattern is April Tag. By utilizing high computingperformance of a modern high-speed central processing unit and a generalpurpose graphics processing unit, pattern decoding and positioncalculation thereof may be performed in real time (for example, lessthan 1/60s). To avoid user dizziness and provide a high refresh rate(such as 60 Hz), this real-time calculation is necessary in the VR andAR (augmented reality) systems.

In addition, there are other means for improving the tracking quality,for example, a method of prediction, which is used for predicting aresult of next tracking and increasing a response speed of the system.Or a smoothing and buffering method may be permissible, which is usedfor reducing an error caused by rapid movement of the object or poorquality of other photographed pictures due to a tracking error throughan algorithm for filtering or smoothing noises using mobile data of aprevious segment of the object.

A photo acquired by the camera may be used for recognizing a posture anda position of a hand.

Use of a hand is the most natural method for interaction between theuser and the external environment. This system includes a hand positiontracking and gesture recognizing module.

An RGB camera may be used for capturing a 2D scenario. Because of asuitable price, the RGB camera is the most widely used in the market.The hand position and the tracking and recognition generally need to besegmented and positioned via background. However, it is difficult toensure to acquire highly reliable hand position and gesture onlyaccording to information in the 2D photo. This is because of: 1) thehand being a non-rigid object; 2) skin colors of different races andskin textures of different crowds; 3) lamplight conditions in differentenvironments; and 4) interferences of background objects in differentenvironments.

Nevertheless, results in this aspect may be improved by utilizing otherinformation in this system. When the user interacts with a booklet inthis system, one or a series of preset identifiers in the booklet may beobstructed by a moving hand because the hand of the user performsoperation above the booklet, so that an image of the preset identifieris not a complete image. When a plurality of hand candidate regions aredetected in the system, the hand candidate regions may be filtered outby determining whether the image of the obstructed preset identifieroverlaps with different candidate hand positions. This is more possibleto provide a highly reliable tracking result than there being the handposition only and being without preset identifier blocking informationof the booklet.

Meanwhile, in an embodiment, this system includes a model that canrecognize the skin color. Nevertheless, because of different skin colorcharacteristics of various races and different lamplight environments,the user may select to establish a skin color model of his/her own. Theflow is described below.

The user is guided to wear the helmet (HMD) and put the booklet in frontof the helmet camera (for example, putting the booklet on a desk infront or grasping the booklet by the hand). Meanwhile, the systemcalculates the position of the booklet via the code tracking module.

The helmet displays a region above the virtual booklet, and the user isguided to cover the region by the hand and stay for a short period oftime (for example 5 s).

The system may determine whether the hand has stayed on this regionthrough color changes before and after the coverage or by determiningwhether the preset identifier on the region is covered, etc.

After the system examines that the hand covers the region, the colorrange of the hand is acquired via color segmentation. By utilizing thedata, an accurate hand segmentation model according to colors may beestablished. This model is established according to the skin colorcharacteristics of the user and the environment where the user is.Therefore, the accuracy of subsequent hand segmentation can be improved.

FIGS. 7-11 schematically illustrate schematic diagrams of handcalibration procedures according to exemplary embodiments of the presentdisclosure. Referring to FIGS. 7-11:

In FIG. 7, the user is guided to put the hand on the booklet.

In FIG. 8, when the user moves the hand on the booklet, the color of anindication mark changes from red to green.

In FIG. 9, the user is required to rest the hand on the booklet for awhile, and then the calculation process is displayed on the screen.

In FIG. 10, the hand positioning process is to be finished, and the handdisplayed on the screen is white.

In FIG. 11, the hand positioning process is finished, the handsegmentation model based on the color is calibrated.

Furthermore, if the HMD has a RGD-D Camera, the result of handsegmentation and the three-dimensional position thereof may be directlyacquired from an image with depth information.

After acquiring the result of hand segmentation, the system recognizesthe gesture. The gesture recognition may be implemented via technologiessuch as a contour line extraction method or a graphics classifier. Afterrecognizing the gesture, the system may trigger a relative command, forexample, page flipping or opening command.

To differentiate between a gesture supported by the system and a generalhand action of the user, an activation sequence may be defined. It isstipulated that the user needs to activate the gesture command accordingto the designed activation sequence. From the screen of the virtualhelmet, the user may obtain a feedback of the activation process.

FIGS. 12-15 schematically illustrate schematic diagrams of procedures ofinitiating or starting an object with gesture commands according toexemplary embodiments of the present disclosure.

In the page center of the virtual booklet, the user may see an operationicon playing a role of indication, such as a thumb icon or icons ofother shapes. When the virtual booklet corresponds to the physicalbooklet, the user may move a finger to the center of the booklet toactivate an action. At this moment, the position of the finger of theuser may be indicated by displaying a green dot.

As an order of activating the gesture command, the user may be firstrequired to stay the finger at the icon position for a period of presettime. For example, a circle or an annulus gradually changing to becomplete is displayed in the icon position on the screen to remind theuser how soon the finger may be removed to activate or start the object.The gesture command will not be activated if the user moves the fingerout of the region of the icon position in this process.

The action will be activated if the user follows the activationprocedure. At this moment, the virtual environment may display an objectcorresponding to the position of finger, which may be, for example, apicture, a photo, video, sound, a 3D stereoscopic object, and so on.

FIGS. 16-21 schematically illustrate schematic diagrams of simulating auser's flipping over procedures according to exemplary embodiments ofthe present disclosure.

Referring to FIG. 16 and FIG. 17, when the user is in a stationarystate, the finger is beyond the region of the booklet. FIG. 17illustrates an image displayed in front of the user's eyes in the stateof FIG. 16.

Referring to FIG. 18 and FIG. 19, when the user wants to flip to anotherpage, the user may move the finger or the hand to the right edge of thebooklet. The page edge being slightly flipped is correspondinglydisplayed on the screen.

Referring to FIG. 20 and FIG. 21, complete activation actions executedby moving, by the user, the finger or the hand to the right in a periodof predetermined time are illustrated.

As illustrated from FIG. 18 to FIG. 21, it may be determined that thefinger or the hand of the user covers up the preset identifier (FIG. 18)at the right edge of the booklet in an initial state and the finger orthe hand of the user covers up the preset identifier (FIG. 21) in themiddle of the booklet in a final state by monitoring an image of thepreset identifier on the booklet, for example, by detecting whether theimage of the preset identifier is complete. It may be determined thatthe finger or the hand of the user moves from right to left by means ofthe position order of the preset identifier covered up. Therefore, apage flipping action from right to left may be performed on the virtualbooklet displayed in the virtual world.

In addition to a booklet in the virtual world, the booklet mentionedabove also may represent other elements or objects. Meanwhile, functionsof the booklet may be further expanded by changing the posture and theoperation of the booklet. These functions may not only be implemented assome additional media (for example, mini-games in magazines, orinteractive experiments in textbooks) in the booklet, but also may beimplemented as an independent unit of a carrier independent of thebooklet. Some examples are enumerated as below:

An operation panel of the interactive experiments in the textbooks (thebooklet may be displayed as an operation plane internally provided withbuttons and other interactive tools);

Mini-games of advertisements in magazines: for example, advertisementsof a car, the booklet may be simulated as a steering wheel (the bookletis moved right and left to simulate swinging the steering wheel rightand left, and the booklet is moved back and forth to simulateacceleration or deceleration of the car).

FIGS. 22-27 schematically illustrate schematic diagrams of procedures ofselecting books by the user according to the exemplary embodiments.

First, the user may enter a book selection interface by using thehand-held preset interactive object (the book) as a selection tool. Theuser may enter the book selection interface by means of a presetspecific gesture operation or by using the buttons or the touch panel onthe HMD or the preset interactive object.

The user may remove the book in the hand to a display position of thetarget book. This step may be implemented by calculating a 3D relativeposition of the user with respect to the book.

The user may convert the posture of the book in the hand to the postureof a virtual target book in the HMD environment, and confirm thisselection intention by holding this action for a period of time. Amarkup interface uses the cover of the target book to replace the coverof the original book.

Corresponding to the above method embodiment, an exemplary embodiment ofthe present disclosure also provides a virtual reality interactionapparatus, which may be employed to execute the above method embodiment.

FIG. 28 is a block diagram of a virtual reality interaction apparatusaccording to an exemplary embodiment of the present disclosure.Referring to FIG. 10, the virtual reality interaction apparatus 300 mayinclude an image capturing module 302, an identifier recognizing module304, a gesture recognizing module 306 and an image output module 308.

The image capturing module 302 may be configured to continuously acquireimage information in front of a user, for example, a camera.

The identifier recognizing module 304 may be configured to recognize apreset interactive object by recognizing a preset identifier in theimage information, wherein a surface of the preset interactive objecthas one or more of the preset identifiers.

The gesture recognizing module 306 may be configured to recognize a handaction of the user according to the image information.

The image output module 308 may be configured to simulate an interactiveaction of the user on the preset interactive object in a virtual realityenvironment according to the hand action and a position of the presetinteractive object.

In an exemplary embodiment, the recognizing a preset interactive objectmay include recognizing the position and/or a surface state of thepreset interactive object.

In an exemplary embodiment, the recognizing a hand action of the usermay include recognizing a hand gesture and/or a hand position.

In an exemplary embodiment, the preset identifier may include virtualcontent information, and the identifier recognizing module is furtherconfigured to recognize the virtual content information.

In an exemplary embodiment, the virtual reality interaction apparatusmay further include a database module, configured to record acorresponding relationship between the preset identifier and the virtualcontent information. The virtual content, displayed on the HMD,corresponding to the ID of the preset identifier on the booklet may beassociated.

Functions of various modules of the virtual reality apparatus 300 havebeen described in detail in the corresponding method embodiments, andthus are not repeated any more herein.

Based on the above embodiments, the present disclosure also provides avirtual reality interaction system, which may include: a virtual realitydisplay apparatus, configured to display a virtual reality image for auser; a preset interactive object, configured to assist in recognizingan interactive action of the user; and the virtual reality interactionapparatus according to the above embodiments.

According to the virtual reality interaction method of the presentdisclosure, a position of a preset interactive object is determined anda hand action of a user is determined by recognizing a preset identifieron the preset interactive object, and the preset interactive object anda relative position relationship between the hand of the user and thepreset interactive object are accurately positioned. In this way, aninteractive action of the user on a real object is accurately simulatedin a virtual reality environment, and user experience of virtual realityis improved.

FIG. 29 schematically illustrates a flowchart of another virtual realityinteraction method according to an exemplary embodiment of the presentdisclosure. As shown in FIG. 29, the virtual reality interaction method400 may include following steps.

In Step S402, image information of a preset interactive object isacquired, wherein a surface of the preset interactive object has apreset identifier.

In an exemplary embodiment, the preset interactive object may be abooklet made from a flexible material, the booklet may include at leastone page, and a surface of the at least one page has a plurality of thepreset identifiers (for example, tags).

In the following embodiments, the booklet is used as a real objectcarrier for operating a VR book, and serves as a major interactive tool.The following embodiments of the present disclosure describe a group ofinteraction on the booklet.

In Step S404, at least one parameter of the preset interactive object isacquired by recognizing the image information corresponding to thepreset identifier.

In an exemplary embodiment, the at least one parameter includes one ormore of bending duration, a bending direction and a bending shape of thepreset interactive object.

In an exemplary embodiment, the acquiring at least one parameter of thepreset interactive object by recognizing the image informationcorresponding to the preset identifier may include: selecting a firstidentifier and a second identifier from a plurality of the presetidentifiers on the at least one page; respectively acquiring a firstnormal vector of the first identifier and a second normal vector of thesecond identifier according to a position coordinate of the firstidentifier and of the second identifier; and determining an includedangle between the first identifier and the second identifier accordingto the first normal vector and the second normal vector.

In Step S406, a preset interactive operation is executed according tothe at least one parameter.

In an exemplary embodiment, the preset interactive operation includesone or more of going to next page, going to previous page, quick flipforward, and quick flip backward, etc.

It is to be noted that although in the embodiments of the presentdisclosure reference is made by taking an example in which the presetinteractive operation is a page flipping control, the present disclosureis not limited thereto. Anyone or more preset interactive operations maybe predefined according to the at least one recognized parameter, forexample, an operation of opening a webpage, or an operation of jumpingto a next video, etc.

In an exemplary embodiment, executing a preset interactive operationaccording to the at least one parameter includes: determining whether acurve of the included angle varying with time falls within a predefinedregion; and executing an operation of going to next page when the curveof the included angle varying with time falls within the predefinedregion.

In an exemplary embodiment, the predefined region is defined by apredefined first curve, a predefined second curve, an intersection pointbetween the first curve and a time axis, and an intersection pointbetween the second curve and the time axis.

In an exemplary embodiment, executing a preset interactive operationaccording to the at least one parameter includes: executing an operationof quick flip forward when the included angle is greater than a firstthreshold and duration exceeds preset time.

In an exemplary embodiment, executing an operation of quick flip forwardwhen the included angle is greater than a first threshold and durationexceeds preset time includes: executing the operation of quick flipforward according to a first animation page flipping speed when theincluded angle is greater than or equal to the first threshold and lessthan a second threshold; and/or executing the operation of quick flipforward according to a second animation page flipping speed when theincluded angle is greater than or equal to the second threshold and lessthan a third threshold; and/or executing the operation of quick flipforward according to a third animation page flipping speed when theincluded angle is greater than or equal to the third threshold and lessthan a fourth threshold; and/or stopping the operation of quick flipforward when the included angle is less than the first threshold.

The above embodiments of the present disclosure are illustrated belowwith reference to FIGS. 30-42. In the following embodiments, anexemplary description is made by taking an example in which pageflipping control is implemented by bending a page of a booklet.

In the embodiments of the present disclosure, reference is made bytaking an example in which the booklet includes a left page and a rightpage, and a plurality of tags are printed or adhered onto oppositeinside surfaces of the left page and the right page. The booklet may bedeformable because it is made of a flexible material. Generally, nodeformation occurs in pages at two sides of the booklet, as shown inFIG. 30. Wherein the tag may be the preset identifier in the previousembodiments.

In a general state of the booklet as shown in FIG. 30, a user may bendthe page with hands. For example, the user incurves the right page ofthe booklet, as shown in FIG. 31; or the user incurves the left page ofthe booklet, as shown in FIG. 32.

The material of the pages at two sides of the booklet has betterelasticity, and thus the page may restore the original state as shown inFIG. 30 after the hands are moved away.

When the page of the booklet is bent, wherein both the degree of bendand the angle of bend thereof may be detected using a camera or otherhardware devices. The bending duration, the bending direction and thebending shape or the like may respectively or collectively define aninteractive operation.

When the booklet bends or deforms, a part of the tags are shielded.Neither positions nor normal vectors of these tags can be obtained byway of direct calculation via a space geometry relationship. Positionsof the tags on the booklet are predetermined, and a larger limitation isimposed on the bending deformation of the booklet, namely, the freedomof deformation is low. Therefore, positions and normal vectors of otherinvisible tags may be inferred through space positions and normalvectors of visible tags on the page.

FIG. 33 schematically illustrates a schematic diagram in which a rightpage of another booklet is incurved according to an exemplary embodimentof the present disclosure.

For example, when the bending deformation as shown in FIG. 33 occurs inthe booklet, three tags on the inner side of the right page cannot becaptured by the camera. However, the space positions and the normalvectors of these invisible tags on the right page may be inferredthrough the positions of visible tags on the left page and the positionsand the normal vectors of visible tags on the right page.

FIG. 34 schematically illustrates a schematic diagram of a normal vectorof a tag of a booklet according to an exemplary embodiment of thepresent disclosure.

In the embodiment of the present disclosure, it is assumed that areal-time image (as shown in FIG. 34) of the booklet held in the handsof the user may be acquired using a front camera of the VR. By using thespace geometry relationship, the space position P=P_(i) ^(camera) andthe normal vector direction N=N_(i) ^(camera) may be calculated out foreach tag in the figure relatively to the camera, wherein the spaceposition of the camera is provided by the VR system. In this way, theposition P_(i) and the normal vector N_(i) of an i-th tag in the worldcoordinate system may be calculated out.

Herein it is assumed that an outward direction of a visible surface on atag is defined as the normal vector direction of the tag, as shown inFIG. 34.

Supposing:

1. The position, of the HMD provided by the VR system, in the worldcoordinate system is P_(HMD) ^(world), and a rotatory posture thereof inthe world coordinate is R_(HMD) ^(world).

2. The position and the posture of the camera relative to the HMD areP_(camera) ^(world) and R_(camera) ^(world) respectively.

3. The position and the posture of the i-th tag relative to the camerarespectively are P_(i) ^(camera) and R_(i) ^(camera) (there are existingalgorithms in the industry for calculating the position and the postureof a tag relative to the camera, which are not unnecessarily elaboratedany more herein).

Based on the above assumptions, the position and the posture of the i-thtag in the world coordinate system respectively are as below:

$\quad\left\{ \begin{matrix}{P_{i} = {{R_{HMD}^{world}\left( {{R_{camera}^{HMD}P_{i}^{camera}} + P_{camera}^{HMD}} \right)} + P_{HMD}^{world}}} \\{R_{i} = {R_{HMD}^{world}R_{camera}^{HMD}R_{i}^{camera}}}\end{matrix} \right.$

after the posture of the i-th tag is obtained, the normal vector N_(i)of the i-th tag may be obtained as below:

N_(i)=R_(i)N₀

where the N₀ is the normal vector when the position and the posture ofthe tag respectively are

$\begin{pmatrix}0 \\0 \\0\end{pmatrix}\mspace{14mu} {and}\mspace{14mu} {\begin{pmatrix}1 & 0 & 0 \\0 & 1 & 0 \\0 & 0 & 1\end{pmatrix}.}$

FIG. 35 schematically illustrates a schematic diagram of a first normalvector in a first identifier and a second normal vector in a secondidentifier according to an exemplary embodiment of the presentdisclosure.

As shown in FIG. 35, two tags as shown in FIG. 35 are selected, whosenormal vectors respectively are

and

.

In the embodiment of FIG. 35, the invisible tag in the middle of thethree tags in the leftmost column on the right page of the booklet isselected as the first identifier, and the visible tag in the middle ofthe three tags in the rightmost column on the right page of the bookletis selected as the second identifier. Advantages of the selections areas below: on the same page, when bending deformation occurs on thispage, the included angle between the normal vectors of two tags in theleftmost column and the rightmost column varies most significantly,which is advantageous to recognizing a degree of variation of theincluded angle.

However, the present disclosure is not limited thereto. In otherembodiments, any two tags on the same page on which bending deformationoccurs may be selected as the first identifier and the secondidentifier.

FIG. 36 schematically illustrates a schematic diagram in which a rightpage bends and deforms with time according to an exemplary embodiment ofthe present disclosure.

In the embodiment as shown in FIG. 36, the degree and direction of theright page bending and deforming with time may be defined by theincluded angle between the first normal vector and the second normalvector selected in FIG. 35.

FIG. 37 schematically illustrates a schematic diagram in which anincluded angle between a first normal vector and a second normal vectorbased on FIG. 36 varies with time.

Supposing the included angle between the first normal vector

and the second normal vector

is denoted as θ. A function of θ varying with time is denoted as θ=θ(t).For example, when the page varies with time just as shown in FIG. 36,the pattern of θ=θ(t) is as shown in FIG. 37.

In some embodiments, an operation of going to next page implemented byincurving the right page just as shown in FIG. 36 may be predefined.However, the present disclosure is not limited thereto.

FIG. 38 schematically illustrates a schematic diagram in which theincluded angle curve as shown in FIG. 37 falls within a predefinedregion according to an embodiment of the present disclosure.

In the embodiment of FIG. 38, an operation of going to next page may bedefined when the curve of θ=θ(t) as shown in FIG. 37 falls within thepredefined region encircled by f₁(t), f₂(t) and a line segment t₁t₂.

The value range of θ_(ε) is [0, 180] degrees. In the embodiment of thepresent disclosure, supposing θ₂₄₉ =10. θ_(ε) denotes it is started todetect whether a subsequent operation satisfies a page flipping actionwhen the page bending angle θ is equal to θ_(ε). No determination ismade when θ<θ_(ε). This is to prevent misoperation from being caused byslight variation of angle of the page resulted from some slight actions.

The t₁ defines the shortest time duration of page bending as a pageflipping operation, whose value range is [0, 60] s. In the embodiment ofthe present disclosure, t₁=0.1 s.

The t₂ defines the longest time duration of page bending as a pageflipping operation, whose value range is [t1, 60] s. In the embodimentof the present disclosure, t₂=1 s.

In the embodiment of the present disclosure, f₁(t) may be as shown inthe following Formula:

${{f_{1}(t)} = {{{- \left( {p_{1} - \theta_{ɛ}} \right)}\left( \frac{2}{t_{1}} \right)^{2}\left( {t - \frac{t_{1}}{2}} \right)^{2}} + p_{1}}},$

where the value range of the p₁ is [θ_(ε), 180] degrees. In theembodiment of the present disclosure, p₁=15 degrees.

In the embodiment of the present disclosure, f₂(t) may be as shown inthe following Formula:

${{f_{2}(t)} = {{{- \left( {p_{2} - \theta_{ɛ}} \right)}\left( \frac{2}{t_{2}} \right)^{2}\left( {t - \frac{t_{2}}{2}} \right)^{2}} + p_{2}}},$

where the value range of the p₂ is [p₁, 180] degrees. In the embodimentof the present disclosure, p₂=45 degrees.

FIG. 39 schematically illustrates another schematic diagram in which anincluded angle curve falls within a predefined region according to anembodiment of the present disclosure.

As shown in FIG. 39, the θ is still the included angle between the firstnormal vector

and the second normal vector

. Region R (OE₁E₂O) is a predefined region. An operation of going tonext page may be triggered when the page bending included angle curve inthe page flipping process is within the Region R and intersects with astraight line E₁E₂ . For example, a curve OME.

In some embodiments, it also may be predefined to trigger an operationof going to previous page when incurving the left page of the bookletsatisfies a certain condition. The above operation of going to next pagemay be referred to for the implementation process of this operation,which is not unnecessarily elaborated any more herein.

FIG. 40 schematically illustrates a schematic diagram of executing anoperation of quick flip forward according to an embodiment of thepresent disclosure.

As shown in FIG. 40, it may be predefined to trigger an operation ofquick flip forward when incurving the right page of the bookletsatisfies a certain condition. In addition, it also maybe predefined toincrease/maintain/decrease a page flipping speed or stop page flippingwhen the angle of incurving the right page satisfies a certaincondition.

In the embodiment of the present disclosure, it is assumed to startcontinuing animation page flipping when θ>θ₁ and the duration exceedst_(b) s, where the value range of the t_(b) may be [0.01, 100] s. In theembodiment of the present disclosure, t_(b)=1 s.

When θ₁<θ<θ₂, it is assumed that the animation page flipping speed is S₁page/s. The value range of the θ₁ maybe [θ₀, 180] degrees, the valuerange of θ₀ may be [0, 180] degrees; and the value range of the θ₂ maybe [θ₁, 180] degrees.

When θ₂<θ<θ₃, it is assumed that the animation page flipping speed is S₂page/s. The value range of the θ₃ maybe [θ₂, 180] degrees.

In the embodiment of the present disclosure, θ₀=10 degrees, θ₁=30degrees, θ₂=60 degrees, and θ₃=90 degrees. However, the presentdisclosure is not limited thereto.

When θ>θ₃, it is assumed that the animation page flipping speed is S₃page/s.

When θ≤θ₁, it is assumed to stop the operation of quick flip forward.

The value range of S₁ may be [1, 100]. The value range of S₂ may be [S₁,100]. The value range of S₃ may be [S₂, 100].

In the embodiment of the present disclosure, S₁=1, S₂=3, and S₃=5.

It is to be noted that the values and the value ranges of the aboveangles, page flipping speeds and page flipping time may be flexiblyadjusted and selected according to requirements for specific applicationscenarios, and not limited to the enumerated values. In addition,triggering or stopping the operation of quick flip forward andincreasing, maintaining or decreasing a page flipping speed may be setup according to the specific needs.

FIG. 41 schematically illustrates a schematic diagram of an includedangle curve when executing the operation of quick flip forward accordingto an embodiment of the present disclosure.

Based on what is shown in FIG. 40, it is assumed that the included angleθ=θ(t) between the normal vectors of two selected tags is as shown inFIG. 41.

when it is at the moment of t₁, θ>θ₁, and after the duration t_(b)=1 s,the quick flip animation is start at the moment of t_(s)=t₁₊t_(b).

When it is from t_(s) to t₂, θ₁<θ<θ₂, and the animation page flippingspeed is 1 page/s.

When it is from t₂ to t₃, θ₂<θ<θ₃, and the animation page flipping speedis 3 page/s.

When it is from t₃ to t₄, θ>θ₃, and the animation page flipping speed is5 page/s.

When it is from t₄ to t₅, θ₂<θ<θ₃, and the animation page flipping speedis 3 page/s.

When it is from t₅ to t₆, θ₁<θ<θ₂, and the animation page flipping speedis 1 page/s.

When it is from t₆ to t₇, θ₂<θ<θ₃, and the animation page flipping speedis 3 page/s.

When it is from t₇ to t₈, θ>θ₃, and the animation page flipping speed is5 page/s.

When it is from t₈ to t₉, θ₂<θ<θ₃, and the animation page flipping speedis 3 page/s.

When it is from t₉ to t₁₀, θ₁<θ<θ₂, and the animation page flippingspeed is 1 page/s.

When t>t₁₀, θ<θ₁, and the animation page flipping is stopped.

FIG. 42 schematically illustrates a schematic diagram of anotherincluded angle curve when executing the operation of quick flip forwardaccording to an embodiment of the present disclosure;

As shown in FIG. 42, when θ₀<θ<θ₁, the animation page flipping speed maybe 1 page/s.

When θ₁<θ<θ₂, the animation page flipping speed may be 3 page/s.

When θ₂<θ<θ₃, the animation page flipping speed may be 5 page/s.

When θ≤θ₀, the animation page flipping may be stopped.

In some other embodiments, it also may be defined to trigger anoperation of quick flip backward when bending the left page of thebooklet satisfies a preset condition. The above operation of quick flipforward may be referred to for the specific implementation of thisoperation, which is not unnecessarily elaborated any more herein.

FIG. 43 schematically illustrates a block diagram of another virtualreality interaction apparatus according to an exemplary embodiment ofthe present disclosure; and

As shown in FIG. 43, the virtual reality interaction apparatus 500 mayinclude an image capturing module 502, a parameter acquiring module 504,and an interaction execution module 506.

The image capturing module 502 may be configured to acquire imageinformation of a preset interactive object, a surface of the presetinteractive object having a preset identifier.

The parameter acquiring module 504 maybe configured to acquire at leastone parameter of the preset interactive object by recognizing the imageinformation corresponding to the preset identifier.

The interaction execution module 506 may be configured to execute apreset interactive operation according to the at least one parameter.

Functions of various modules of the virtual reality apparatus 500 havebeen described in detail in the corresponding method embodiments, andthus are not repeated any more herein.

FIG. 45 is a flowchart of a virtual reality interaction method accordingto another embodiment of the present disclosure. Referring to FIG. 45,the virtual reality interaction method 4500 may include following steps:

Step S4502: rendering content including an object;

Step S4504: tracking the object in the content; and

Step S4506: performing gesture tracking for supporting gesture controlon the object.

According to the virtual reality interaction method 4500 of the presentdisclosure, an AR platform, which allows scientists, developers andusers to research and develop AR components and assembly them intoapplications to service distributed users, is provided. In anembodiment, an application programing interface API framework isprovided to allow a user to define at least one of the content, thetracking module and the interactivity module.

More specifically, an AR application typically consists of threecomponents: content, object tracking, object interactivity. The ARplatform provides standardized APIs or glues to allow modules of thesethree components developed by third parties to inter-operate with oneanother. Content providers can upload any 2D, 3D, VR content, or providephysical objects. Scientists and developers can supply object trackingmodules to track either physical objects or virtual objects in 2D/3D/VRcontent. Developers can also provide interactivity modules to performe.g., eye tracking and hand tracking to support eye or gesture control.Application developers or system integrators can put together anapplication or a system by composing modules of three components intoscenarios.

In an embodiment, Step S4502 includes rendering a virtual book on apiece of card stock to allow a user to select and browse content. Forexample, FIG. 19 shows a virtual book rendered on the physical cardstock, and FIG. 18 shows a piece of card stock as a physical object. Tointeract with a virtual book via the physical card stock, developers canupload their devised gesture modules such as pin-point, flip a page,copy-and-paste and support desired interactivity. In general, a physicalobject is turned into or overlaid with a virtual object when viewedthrough an HMD for interactivity. Content, tracking modules, andinteractivity modules can be supplied by anyone to the AR platform, andthen they can be integrated into scenarios to support education,training, and tourism, to name just a few verticals.

To enable module integration, the AR platform provides the followingbuilding blocks, and again, third parties can upload their own.

In an embodiment, the AR platform provides a glue layer via APIs.Through a PPT-like interface, an application developer can mark on aselected content some coordinates for user interactions. For example,some buttons can be marked on a museum art piece. Next, on thesebuttons, HCI (Human-Computer Interaction) actions are defined. Somebuttons can be activated by pointing, and some by pressing, and some byswiping. Once these actions have been defined, an application can begenerated.

In an embodiment, the AR platform further provides function of markertracking. The objects may be accurately and timely to perform pose(position and orientation) estimation. One possible implementation is toprint markers on some 2D/3D physical objects. Another possibilityimplementation is to track objects directly by recognizing thecharacteristics of the tracked objects. The present disclosure does notdictate a particular implementation. Instead, the AR platform providesstandard APIs to permit an object to be tracked by any selected trackingmodule.

In an embodiment, the AR platform further provides HCI means and aneditor. A user is enabled to interact with the AR platform using variousHCI means, such as eye gazing, hand gestures, and via a controller.Also, the AR platform permits any HCI module to be uploaded and used byan application. Moreover, the editor is provided by AR platform tocompose scenarios using content, tracking modules, and HCI modules.

FIG. 46 schematically illustrates a block diagram of a virtual realityinteraction system according to an exemplary embodiment of the presentdisclosure. As shown in FIG. 46, the virtual reality interaction system4600 may include a content display 4601, a tracking module 4602 and aninteractivity module 4603.

In an embodiment, the content display 4601 is configured to rendercontent including an object, the tracking module 4602 is configured totrack the object in the content, and the interactivity module 4603 isconfigured to perform gesture tracking for supporting gesture control onthe object. Furthermore, an API framework is configured to allow a userto define at least one of the content, the tracking module and theinteractivity module.

In an embodiment, the object includes at least one of a virtual objectand a physical object.

In an embodiment, the content display is configured to enable thephysical object to be turned into or overlaid with a virtual object whenviewed through a head-mounted display HMD.

In an embodiment, the content display is configured to render a virtualbook on a piece of card stock.

In an embodiment, the gesture control includes pin-point, flip a pageand copy-and-paste.

In an embodiment, the API framework is further configured to provide aglue layer above the content, the glue layer including a marker arrangedcorresponding to a coordinate of the content, the marker beingconfigured to be activated for user interaction.

In an embodiment, the tracking module is configured to track the objectby recognizing the marker or recognizing characteristic of the object.

Functions of various modules of the virtual reality apparatus 500 havebeen described in detail in the corresponding method embodiments, andthus are not repeated any more herein.

It is to be noticed that although a plurality of modules or units of thedevice for action execution have been mentioned in the above detaileddescription, this partition is not compulsory. Actually, according tothe embodiment of the present disclosure, features and functions of twoor more modules or units as described above may be embodied in onemodule or unit. Reversely, features and functions of one module or unitas described above may be further embodied in more modules or units.

In addition, steps of the method in the present disclosure are describedin a particular order in the accompanying drawings. However, this doesnot require or imply to execute these steps necessarily according to theparticular order, or this does not mean that the expected result cannotbe implemented unless all the shown steps are executed. Additionally oralternatively, some steps may be omitted, a plurality of steps may becombined into one step for execution, and/or one step may be decomposedinto a plurality of steps for execution.

FIG. 44 schematically illustrates a schematic structural diagram of anelectronic device according to an exemplary embodiment of the presentdisclosure.

Referring to FIG. 44, a schematic structural diagram of a computersystem 600 adapted to implement the embodiment of the present disclosureis shown. The electronic device as shown in FIG. 44 is merely anexample, and no limitation should be imposed on functions or scope ofuse of the embodiment of the present disclosure.

As shown in FIG. 44, the electronic device 600 includes a centralprocessing unit (CPU) 601, which may execute various appropriate actionsand processes in accordance with a program stored in a read-only memory(ROM) 602 or a program loaded into a random access memory (RAM) 603 froma storage portion 608. The RAM 603 also stores various programs and datarequired by operations of the system 600. The CPU 601, the ROM 602 andthe RAM 603 are connected to each other through a bus 604. Aninput/output (I/O) interface 605 is also connected to the bus 604.

The following components are connected to the I/O interface 605: aninput portion 606 including a keyboard, a mouse etc.; an output portion607 comprising a cathode ray tube (CRT), a liquid crystal display device(LCD), a speaker etc.; a storage portion 608 including a hard disk andthe like; and a communication portion 609 comprising a network interfacecard, such as a LAN card and a modem. The communication portion 609performs communication processes via a network, such as the Internet. Adriver 610 is also connected to the I/O interface 605 as required. Aremovable medium 611, such as a magnetic disk, an optical disk, amagneto-optical disk, and a semiconductor memory, may be installed onthe driver 610, to facilitate the retrieval of a computer program fromthe removable medium 611, and the installation thereof on the storageportion 608 as needed.

In particular, according to an embodiment of the present disclosure, theprocess described above with reference to the flow chart may beimplemented in a computer software program. For example, an embodimentof the present disclosure includes a computer program product, whichcomprises a computer program that is tangibly embedded in acomputer-readable medium. The computer program comprises program codesfor executing the method as illustrated in the flowchart. In such anembodiment, the computer program may be downloaded and installed from anetwork via the communication portion 609, and/or may be installed fromthe removable media 611. The computer program, when executed by the CPU601, implements the functions as defined by the system of the presentdisclosure.

The flowcharts and block diagrams in the figures illustratearchitectures, functions and operations that may be implementedaccording to the system, the method and the computer program product ofthe various embodiments of the present disclosure. In this regard, eachblock in the flow charts and block diagrams may represent a module, aprogram segment, or a code portion. The module, the program segment, orthe code portion comprises one or more executable instructions forimplementing the specified logical function. It should be noted that, insome alternative implementations, the functions denoted by the blocksmay occur in a sequence different from the sequences shown in thefigures. For example, in practice, two blocks in succession may beexecuted, depending on the involved functionalities, substantially inparallel, or in a reverse sequence. It should also be noted that, eachblock in the block diagrams or the flowcharts and/or a combination ofthe blocks in the block diagrams or the flowcharts may be implemented bya dedicated hardware-based system executing specified functions oroperations, or by a combination of a dedicated hardware and computerinstructions.

The units or modules involved in the embodiments of the presentdisclosure may be implemented by way of software or hardware. Thedescribed units also may be provided in a processor, for example,described as: a processor, comprising a sending unit, an acquiring unit,a determining unit, and a first processing unit. Names of these unitsare not considered as a limitation to the units in some cases. Forexample, the sending unit also may be described as “a unit for sending apicture acquiring request to a connected server”.

In another aspect, the present disclosure further provides acomputer-readable medium. The computer-readable medium may be the mediumincluded in the device as described in the above embodiment, or astand-alone medium which has not been assembled into the device. Thecomputer-readable medium hosts one or more programs. When the one ormore programs are executed by the device, the device is caused to:continuously acquire image information in front of a user; recognize apreset interactive object by recognizing a preset identifier in theimage information, a surface of the preset interactive object having oneor more of the preset identifiers; recognize a hand action of the useraccording to the image information; and simulate an interactive actionof the user on the preset interactive object in a virtual realityenvironment according to the hand action and a position of the presetinteractive object.

In another aspect, the present disclosure further provides anothercomputer-readable medium. The computer-readable medium may be the mediumincluded in the device described in the above embodiments, or astand-alone medium which has not been assembled into the device. Thecomputer-readable medium hosts one or more programs. When the one ormore programs are executed by the device, the device is caused to:acquire image information of a preset interactive object, a surface ofthe preset interactive object having a preset identifier; acquire atleast one parameter of the preset interactive object by recognizing theimage information corresponding to the preset identifier; and execute apreset interactive operation according to the at least one parameter.

Other embodiments of the present disclosure will be apparent to thoseskilled in the art from consideration of the specification and practiceof the present disclosure disclosed here. This application is intendedto cover any variations, uses, or adaptations of the present disclosurefollowing the general principles thereof and including such departuresfrom the present disclosure as come within known or customary practicein the art. It is intended that the specification and embodiments beconsidered as exemplary only, with a true scope and spirit of thepresent disclosure being indicated by the appended claims.

Those skilled in the art will readily observe that numerousmodifications and alterations of the device and method may be made whileretaining the teachings of the invention. Accordingly, the abovedisclosure should be construed as limited only by the metes and boundsof the appended claims.

What is claimed is:
 1. A virtual reality interaction method, comprising:rendering content comprising an object; tracking the object in thecontent; and performing gesture tracking for supporting gesture controlon the object; wherein an application programing interface API frameworkis provided to allow a user to define at least one of the content, thetracking module and the interactivity module.
 2. The method according toclaim 1, wherein the object comprises at least one of a virtual objectand a physical object.
 3. The method according to claim 2, wherein therendering content comprising an object comprises: enabling the physicalobject to be turned into or overlaid with a virtual object when viewedthrough a head-mounted display HMD.
 4. The method according to claim 2,wherein the rendering content comprising an object comprises: renderinga virtual book on a piece of card stock.
 5. The method according toclaim 4, wherein the gesture control comprises pin-point, flip a pageand copy-and-paste.
 6. The method according to claim 1, wherein the APIframework is further configured to provide a glue layer above thecontent, the glue layer comprising a marker arranged corresponding to acoordinate of the content, the marker being configured to be activatedfor user interaction.
 7. The method according to claim 6, wherein thetracking the object in the content comprises: tracking the object byrecognizing the marker or recognizing characteristic of the object.
 8. Avirtual reality interaction system, comprising: a content displayconfigured to render content comprising an object; a tracking moduleconfigured to track the object in the content; an interactivity moduleconfigured to perform gesture tracking for supporting gesture control onthe object; and an application programing interface API frameworkconfigured to allow a user to define at least one of the content, thetracking module and the interactivity module.
 9. The system according toclaim 8, wherein the object comprises at least one of a virtual objectand a physical object.
 10. The system according to claim 9, wherein thecontent display is configured to enable the physical object to be turnedinto or overlaid with a virtual object when viewed through ahead-mounted display HMD.
 11. The system according to claim 9, whereinthe content display is configured to render a virtual book on a piece ofcard stock.
 12. The system according to claim 11, wherein the gesturecontrol comprises pin-point, flip a page and copy-and-paste.
 13. Thesystem according to claim 8, wherein the API framework is furtherconfigured to provide a glue layer above the content, the glue layercomprising a marker arranged corresponding to a coordinate of thecontent, the marker being configured to be activated for userinteraction.
 14. The system according to claim 13, wherein the trackingmodule is configured to track the object by recognizing the marker orrecognizing characteristic of the object.
 15. An electronic device,comprising: one or more processors; and a storage apparatus, configuredto store one or more programs; when the one or more programs areexecuted by the one or more processors, the one or more processors arecaused to implement: rendering content comprising an object; trackingthe object in the content; and performing gesture tracking forsupporting gesture control on the object; wherein an applicationprograming interface API framework is provided to allow a user to defineat least one of the content, the tracking module and the interactivitymodule.
 16. The method according to claim 1, wherein the objectcomprises at least one of a virtual object and a physical object. 17.The method according to claim 16, wherein the one or more processorscaused to implement rendering content comprising an object are furthercaused to implement: enabling the physical object to be turned into oroverlaid with a virtual object when viewed through a head-mounteddisplay HMD.
 18. The method according to claim 16, wherein the APIframework is further configured to provide a glue layer above thecontent, the glue layer comprising a marker arranged corresponding to acoordinate of the content, the marker being configured to be activatedfor user interaction.
 19. The method according to claim 18, wherein theone or more processors caused to implement tracking the object in thecontent are further caused to implement: tracking the object byrecognizing the marker or recognizing characteristic of the object. 20.A computer-readable medium containing instructions for performing themethod of claim 1.